Estimating Tail Probabilities Research Articles

Measuring Discrete Risks on Infinite Domains: Theoretical Foundations, Conditional Five Number Summaries, and Data Analyses

To accommodate numerous practical scenarios, in this article we extend statistical inference for smoothed quantile estimators from finite domains to infinite domains. We accomplish the task with the help of a newly designed truncation methodology for discrete loss distributions with infinite domains. A simulation study illustrates the methodology in the case of several distributions, such as Poisson, negative binomial, and their zero-inflated versions, which are commonly used in the insurance industry to model claim frequencies. Additionally, we propose a very flexible bootstrap-based approach for use in practice. Using automobile accident data and their modifications, we compute what we have termed the conditional five number summary (C5NS) for the tail risk and construct confidence intervals for each of the five quantiles making up C5NS and then calculate the tail probabilities. The results show that the smoothed quantile approach classifies the tail riskiness of portfolios not only more accurately but also produces lower coefficients of variation in the estimation of tail probabilities than those obtained using the linear interpolation approach.

ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions

Outlier detection refers to the identification of data points that deviate from a general data distribution. Existing unsupervised approaches often suffer from high computational cost, complex hyperparameter tuning, and limited interpretability, especially when working with large, high-dimensional datasets. To address these issues, we present a simple yet effective algorithm called ECOD (Empirical-Cumulative-distribution-based Outlier Detection), which is inspired by the fact that outliers are often the "rare events" that appear in the tails of a distribution. In a nutshell, ECOD first estimates the underlying distribution of the input data in a nonparametric fashion by computing the empirical cumulative distribution per dimension of the data. ECOD then uses these empirical distributions to estimate tail probabilities per dimension for each data point. Finally, ECOD computes an outlier score of each data point by aggregating estimated tail probabilities across dimensions. Our contributions are as follows: (1) we propose a novel outlier detection method called ECOD, which is both parameter-free and easy to interpret; (2) we perform extensive experiments on 30 benchmark datasets, where we find that ECOD outperforms 11 state-of-the-art baselines in terms of accuracy, efficiency, and scalability; and (3) we release an easy-to-use and scalable (with distributed support) Python implementation for accessibility and reproducibility.

Saddlepoint approximations to score test statistics in logistic regression for analyzing genome-wide association studies.

We investigate saddlepoint approximations of tail probabilities of the score test statistic in logistic regression for genome-wide association studies. The inaccuracy in the normal approximation of the score test statistic increases with increasing imbalance in the response and with decreasing minor allele counts. Applying saddlepoint approximation methods greatly improve the accuracy, even far out in the tails of the distribution. By using exact results for a simple logistic regression model, as well as simulations for models with nuisance parameters, we compare double saddlepoint methods for computing two-sided -values and mid- -values. These methods are also compared to a recent single saddlepoint procedure. We investigate the methods further on data from UK Biobank with skin and soft tissue infections as phenotype, using both common and rare variants.

Comparison Between Bayesian and Frequentist Tail Probability Estimates

Tail probability plays an important part in the extreme value theory. Sometimes the conclusions from two approaches for estimating the tail probability of extreme events, the Bayesian and the frequentist methods, can differ a lot. In 1999, a rainfall that caused more than 30,000 deaths in Venezuela was not captured by the simple frequentist extreme value techniques. However, this catastrophic rainfall was not surprising if the Bayesian inference was used to allow for parameter uncertainty and the full available data was exploited [4]. In this paper, we investigate the reasons that the Bayesian estimator of the tail probability is always higher than the frequentist estimator. Sufficient conditions for this phenomenon are established both by using Jensen’s Inequality and by looking at Taylor series approximations, both of which point to the convexity of the distribution function.

Tail probability estimates of continuous-time simulated annealing processes

<p style='text-indent:20px;'>We study the convergence rate of a continuous-time simulated annealing process <inline-formula><tex-math id="M1">\begin{document}$ (X_t; \, t \ge 0) $\end{document}</tex-math></inline-formula> for approximating the global optimum of a given function <inline-formula><tex-math id="M2">\begin{document}$ f $\end{document}</tex-math></inline-formula>. We prove that the tail probability <inline-formula><tex-math id="M3">\begin{document}$ \mathbb{P}(f(X_t) > \min f +\delta) $\end{document}</tex-math></inline-formula> decays polynomial in time with an appropriately chosen cooling schedule of temperature, and provide an explicit convergence rate through a non-asymptotic bound. Our argument applies recent development of the Eyring-Kramers law on functional inequalities for the Gibbs measure at low temperatures.</p>

Estimating Tail Probabilities of Random Sums of Phase-Type Scale Mixture Random Variables

We consider the problem of estimating tail probabilities of random sums of scale mixture of phase-type distributions—a class of distributions corresponding to random variables which can be represented as a product of a non-negative but otherwise arbitrary random variable with a phase-type random variable. Our motivation arises from applications in risk, queueing problems for estimating ruin probabilities, and waiting time distributions, respectively. Mixtures of distributions are flexible models and can be exploited in modelling non-life insurance loss amounts. Classical rare-event simulation algorithms cannot be implemented in this setting because these methods typically rely on the availability of the cumulative distribution function or the moment generating function, but these are difficult to compute or are not even available for the class of scale mixture of phase-type distributions. The contributions of this paper are that we address these issues by proposing alternative simulation methods for estimating tail probabilities of random sums of scale mixture of phase-type distributions which combine importance sampling and conditional Monte Carlo methods, showing the efficiency of the proposed estimators for a wide class of scaling distributions, and validating the empirical performance of the suggested methods via numerical experimentation.

Gel’fand–Yaglom type equations for calculating fluctuations around instantons in stochastic systems

In recent years, instanton calculus has successfully been employed to estimate tail probabilities of rare events in various stochastic dynamical systems. Without further corrections, however, these estimates can only capture the exponential scaling. In this paper, we derive a general, closed form expression for the leading prefactor contribution of the fluctuations around the instanton trajectory for the computation of probability density functions of general observables. The key technique is applying the Gel’fand–Yaglom recursive evaluation method to the suitably discretized Gaussian path integral of the fluctuations, in order to obtain matrix evolution equations that yield the fluctuation determinant. We demonstrate agreement between these predictions and direct sampling for examples motivated from turbulence theory.

Probabilistic robustness estimates for feed-forward neural networks

Robustness of deep neural networks is a critical issue in practical applications. In the general case of feed-forward neural networks (including convolutional deep neural network architectures), under random noise attacks, we propose to study the probability that the output of the network deviates from its nominal value by a given threshold. We derive a simple concentration inequality for the propagation of the input uncertainty through the network using the Cramer–Chernoff method and estimates of the local variation of the neural network mapping computed at the training points. We further discuss and exploit the resulting condition on the network to regularize the loss function during training. Finally, we assess the proposed tail probability estimates empirically on various public datasets and show that the observed robustness is very well estimated by the proposed method.

Cramér-type moderate deviation theorems for nonnormal approximation

A Cram\'er-type moderate deviation theorem quantifies the relative error of the tail probability approximation. It provides theoretical justification when the limiting tail probability can be used to estimate the tail probability under study. Chen Fang and Shao (2013) obtained a general Cram\'er-type moderate result using Stein's method when the limiting was a normal distribution. In this paper, Cram\'er-type moderate deviation theorems are established for nonnormal approximation under a general Stein identity, which is satisfied via the exchangeable pair approach and Stein's coupling. In particular, a Cram\'er-type moderate deviation theorem is obtained for the general Curie--Weiss model and the imitative monomer-dimer mean-field model.

Estimation of Tail Probabilities by Repeated Augmented Reality.

Synthetic data, when properly used, can enhance patterns in real data and thus provide insights into different problems. Here, the estimation of tail probabilities of rare events from a moderately large number of observations is considered. The problem is approached by a large number of augmentations or fusions of the real data with computer-generated synthetic samples. The tail probability of interest is approximated by subsequences created by a novel iterative process. The estimates are found to be quite precise.

Inequalities for acceptable noncommutative random variables

We introduce the notion of acceptable noncommutative random variables and investigate their essential properties. More precisely, we provide several efficient estimation of tail probabilities of sums of noncommutative random variables under some mild conditions. Moreover, we investigate the complete convergence of a sequence of the form [Formula: see text] in which [Formula: see text] is a sequence of acceptable noncommutative random variables. Our results extend some classical inequalities as well.

Model selection in radon data fusion

Abstract Fitting parametric models or the use of the empirical cumulative distribution function are problematic when it comes to the estimation of tail probabilities from small samples. A possible remedy is to fuse or combine the small samples with additional data from external sources and base the inference on the so called density ratio model with variable tilt functions, which widens the support of the estimated distribution of interest. This approach is illustrated using residential radon concentration data collected from western Pennsylvania.

SOME NEW BOUNDS AND APPROXIMATIONS ON TAIL PROBABILITIES OF THE POISSON AND OTHER DISCRETE DISTRIBUTIONS

AbstractIn this paper, we discuss new bounds and approximations for tail probabilities of certain discrete distributions. Several different methods are used to obtain bounds and/or approximations. Excellent upper and lower bounds are obtained for the Poisson distribution. Excellent approximations (and not bounds necessarily) are also obtained for other discrete distributions. Numerical comparisons made to previously proposed methods demonstrate that the new bounds and/or approximations compare very favorably. Some conjectures are made.

When Less Is More: On the Use of Historical Yield Data with Application to Rating Area Crop Insurance Contracts

AbstractCrop insurance is the cornerstone program of domestic farm policy in most developed countries. Although most countries’ rating methodology corrects for time-varying movements in the first two moments, it is unclear whether using the entire yield series remains appropriate.  We use distributional tests and an out-of-sample retain-cede rating game to answer whether governments/insurers should historically trim yields in estimating their premium rates. Despite small sample sizes and the need to estimate tail probabilities, the historical data appear to be sufficiently different such that trimming is justified.

Saddlepoint approximations for the distribution of some robust estimators of the variogram

In this paper, we obtain a saddlepoint approximation for the small sample distribution of several variogram estimators such as the classical Matheron’s estimator, some M-estimators like the robust Huber’s variogram estimator, and also the $$\alpha $$-trimmed variogram estimator. The tail probability approximation obtained is very accurate even for small sample sizes. In the approximations we consider that the observations follow a distribution close to the normal, specifically, a scale contaminated normal model. To obtain the approximations we transform the original observations into a new ones, which can be considered independent if a linearized variogram can be accepted as model for them. To check this, a goodness of fit test for a variogram model is defined in the last part of the paper.

Random weighting-based quantile estimation via importance resampling

This paper presents a new method to estimate the quantiles of generic statistics by combining the concept of random weighting with importance resampling. This method converts the problem of quantile estimation to a dual problem of tail probabilities estimation. Random weighting theories are established to calculate the optimal resampling weights for estimation of tail probabilities via sequential variance minimization. Subsequently, the quantile estimation is constructed by using the obtained optimal resampling weights. Experimental results on real and simulated data sets demonstrate that the proposed random weighting method can effectively estimate the quantiles of generic statistics.

On a new class of score functions to estimate tail probabilities of some stochastic processes with adaptive multilevel splitting.

We investigate the application of the adaptive multilevel splitting algorithm for the estimation of tail probabilities of solutions of stochastic differential equations evaluated at a given time and of associated temporal averages. We introduce a new, very general, and effective family of score functions that is designed for these problems. We illustrate its behavior in a series of numerical experiments. In particular, we demonstrate how it can be used to estimate large deviations rate functionals for the longtime limit of temporal averages.

Nonparametric estimation of multivariate tail probabilities and tail dependence coefficients

We propose three methods for estimating the joint tail probabilities based on a d-variate copula with dimension d≥2. For the first two methods, we use two different tail expansions of the copula which are valid under mild regularity conditions. We estimate the coefficients of these expansions using the maximum likelihood approach with appropriate data beyond a threshold in the tail. For the third method, we propose a family of tail-weighted measures of multivariate dependence and use these measures to estimate the coefficients of the second tail expansion using regression. This expansion is then used to estimate the joint tail probabilities when the empirical probabilities cannot be used because of lack of data in the tail. The three proposed methods can also be used to estimate tail dependence coefficients of a multivariate copula. Simulation studies are used to indicate when the methods give more accurate estimates of the tail probabilities and tail dependence coefficients. We apply the proposed methods to analyze tail properties of a data set of financial returns.

Hybrid importance sampling Monte Carlo approach for yield estimation in circuit design

The dimension of transistors shrinks with each new technology developed in the semiconductor industry. The extreme scaling of transistors introduces important statistical variations in their process parameters. A large digital integrated circuit consists of a very large number (in millions or billions) of transistors, and therefore the number of statistical parameters may become very large if mismatch variations are modeled. The parametric variations often cause to the circuit performance degradation. Such degradation can lead to a circuit failure that directly affects the yield of the producing company and its fame for reliable products. As a consequence, the failure probability of a circuit must be estimated accurately enough. In this paper, we consider the Importance Sampling Monte Carlo method as a reference probability estimator for estimating tail probabilities. We propose a Hybrid ISMC approach for dealing with circuits having a large number of input parameters and provide a fast estimation of the probability. In the Hybrid approach, we replace the expensive to use circuit model by its cheap surrogate for most of the simulations. The expensive circuit model is used only for getting the training sets (to fit the surrogates) and near to the failure threshold for reducing the bias introduced by the replacement.

Parametric models for biomarkers based on flexible size distributions.

Recent advances in social science surveys include collection of biological samples. Although biomarkers offer a large potential for social science and economic research, they impose a number of statistical challenges, often being distributed asymmetrically with heavy tails. Using data from the UK Household Panel Survey, we illustrate the comparative performance of a set of flexible parametric distributions, which allow for a wide range of skewness and kurtosis: the four‐parameter generalized beta of the second kind (GB2), the three‐parameter generalized gamma, and their three‐, two‐, or one‐parameter nested and limiting cases. Commonly used blood‐based biomarkers for inflammation, diabetes, cholesterol, and stress‐related hormones are modelled. Although some of the three‐parameter distributions nested within the GB2 outperform the latter for most of the biomarkers considered, the GB2 can be used as a guide for choosing among competing parametric distributions for biomarkers. Going “beyond the mean” to estimate tail probabilities, we find that GB2 performs fairly well with some disparities at the very high levels of glycated hemoglobin and fibrinogen. Commonly used linear models are shown to perform worse than almost all the flexible distributions.